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ABSTRACT 
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production of generalizable rules which explain the influences of 
social variables, individual differences, and cognitive style on the 
paths taken through the instruction. Descriptive and inferential 
approaches to analyzing interactive media audit trails are 
identified, and some of the issues and concerns raised by such 
investigation are discussed. The paper concludes with a consideration 
of research opportunities in interactive technologies of instruction 
that such analysis would promote. Samples of five methods of 
representing audit trails are appended. (9 references) (GL) 
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Ii. traditional (linear) media, such as audio 
materials, video or film, or combinations 
of still pictures and audio, arty learner 
going through the materials Is offered 
essentially the same learning experience as 
any other leamer going through tlie same 
materials (given reasonably similar levels 
of motivation). As a consequence, instruc- 
tional designers have been able to assume a 
certain constancy to their instruction. 
Now, however, the proliferation of interac- 
tive t'i^chnologies and hypermedia has given 
instructional designers the capability of 
producing materials which provide for a 
great deal of choice on the part of learners 
in determining their paths through the 
instruction. This means that two learners 
may be able to traverse the same instruc- 
tional materials, perhaps even achieve the 
same objectives, yet not share a common 



experience. At minimum, the degree of 
commonality in tht experience Is reduced 
over that experienced with the more linear 
media. While obviously having a liberaliz- 
ing (some would say humanizing) effect on 
instruction, the advent of interactive and 
hypermediated technologies brings with 
them a host of new questions for Instruc- 
tional designers, as well Among these is 
the determination of the effects of taking 
different paths through instruction. To 
address such a question requires that there 
be some w^ of recording and analyzing the 
instructional path taken hy each leamer. 
We refer to that record of the path as the 
audit trail (after M. W. Petruk, personal 
communication. February 7, 1990), 

It is a relatively simple matter to collect 
information about an indlvlduars path, 
using contemporary computer-based 
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instruction authoring environments (e.g., 
Authorvvare Professional or Course of 
Action) or program development tools such 
as Macintosh's HyperCard. Learners* 
responses can be trapped and recorded at 
each decision point or node. Of course, the 
descriptive data captvircd can provide very 
useful formative evaluation information. 
However, we speculate that there are addi- 
tional purposes to which such information 
can be put purposes that might produce 
generalizable rules about the construction 
of various i)aths through interactive and 
hypennediated instruction. In order to 
generate such rules, however, questions of 
this type mist be asked and answeix.-d: Can 
we confidently state that one path is signif- 
icant^ different from another path? Can 
we conibine groups of paths and compare 
them to other groups of paths— a necessary 
step if we intend to refine individual differ- 
ences research in interactive media? 

To illustrate the desirability of pursu- 
ing this type of research, consider that 
individuals, and indeed groups of 
individuals, can approach instruction 
differently, and this has traditionally 
been of theoretical interest. For a 
simple example, we might consider an 
Issue like racism. Suppose an instruc- 
tional developer is developing video 
materials for a Canadian (Le., multi- 
cultural) audience. Might the race of 
an on<amera instructor influence 
choices made by a learner within the 
Instructional sequence? What about 
other characteristics— socioeconomic 
status, gender, age, religion? Would 
any or all of these characteristics 
influence the ultimate path taken 
through the instruction? 

As a second example, we could focus 
on arty of a number of individual dif- 



ferences or cognitive styles: consider 
locus of control.* How might Inter- 
nalizers and externalizers differ in 
their approaches to highly organized 
interactive treatments? Would they 
react different^ to, and take different 
paths through, very linear treatments 
and hypermedia treatments? One 
learner may select iht shortest path 
available: another may select every 
available remedial segment in the 
same treatment. Resultant patlis 
would be very different from one 
another, but, short of actually watch- 
ing both individuals progress through 
the materials, how can theee differ- 
ences be e3q)ressed? As interest grows 
in the effectiveness of learner control 
of instruction (e.g., see Hlgginbotham- 
Wheat 1990: L6pez, 1990: Ross, 
Morrison, & ODell, 1990: Steinberg, 
1977), and especially as it broadens 
into learner control of interactive and 
hypermediated instruction, these 
kinds of questions will command 
Increasing Interest. 

In order to explore these types of questions, 
we need some anatytical tools that don't 
appear to have been developed yet. As this 
area of investigation Is still very new. 
prescriptions are not yet at hand. Some 
preliminary work on the subject has been 
done, however, and we attempt in this paper 
to represent in some meaningful ways the 
different paths that individuals may follov/ 
through interactive media treatments. We 
identify both descriptive and Inferential 
approaches to analyzing interactive media 
audit trails, and identify some of the Issues 
and concerns that such investigation 
reveals. We also attempt to identify 
research opportunities in interactive tech- 
nologies of instruction that such analysis 
would promote. 
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The Audit Trail 



An audit trail comprises all the responses 
generated by a learner going through inter- 
active or hypennediated instruction. In 
general terms* audit trails contain words* 
phrases* sentences, and paragraphs that a 
learner types into a computer* as well as a 
record of the "taultlplc choIcc*-llke re- 
sponses made* either firom the keyboard or 
via some other input device* such as a 
mouse or touch-sensitive screen. However* 
to keep this preliminary discussion rela- 
tively simple* we restrict our discussion 
here to the "multiple choIcc"-like re- 
sponses. Hence* we conceive of the audit 
trail as a string of characters (niunerals or 
letters) r^resenting choices made by learn- 
ers as they progress through choice points* 
or nodes, within the instruction. For 
example* suppose the first point at which a 
learner had to make a choice had three dif- 
ferent paths the learner could follow, and 
let us further suppose that she chooses the 
second path. As she proceeds through the 
subsequent instruction, she encounters a 
second ruxle* with two choices* and chooses 
the first of those. Ihe first two characters 
of her audit trail would therefore be 21 (read 
•two-one"* not twenty-one"). If she then 
chooses the first of three paths at the next 
node* and the third of three at the following 
node* her complete audit trail up to this 
point would be 2113. A second leamer 
might have chasen the third path at the 
first node* the first path at the second node* 
the first path at the third node* and the 
second path at the fourth node* to give an 
audit trail of 3 1 12. In creating an ^udit 
trail* the computer would simpty record the 
numerical value assigned to each option 
presented at a decision point* in a vector- 
like arrangement As the number of deci- 
sion points in the treatment increases* 
will the length of the vector; as the com- 
plexity of the path taken by an individual 
increases (e.g.* exploring optional paths as 
opposed to forging straight ahead)* so will 
the length of the vector. 



Some things immediately become obvious: 

1. The meaning of ai^ character in 
the audit trail is dependent for 
meaning upon the character pre- 
ceding it. That is* the character 
*l\ the second character in tae 
first individual's audit trail* does 
not mean the same thirg as the 
character *!* which is the second 
character in the second individ- 
ual's audit trail* because the two 
individuals chose different paths 
at the first node. We wlH refer to 
this phenomenon as the depen- 
dency problem. 

2. A problem is created by repeat visits to 
a node. In zjay particular interactive 
treatment* an individual may intersect 
the same node in the program more 
than once. Stated difierently* the 
leamer may traverse portions of a path 
repeatedly. Thus* if a simple fre- 
quency count is made of visits to a 
node* it is difficult to discriminate* for 
example* between the case in which two 
learners traverse the same path seg- 
ment once each* and one in which a 
single leamer traverses the same path 
segment twice. We wflB refer to this 
problem as the looping problem. The 
looping problem is context-dependent. 
Interactive treatments are typically 
designed to permit repeated visits to 
some portions of instruction but not 
others. A second implication of this 
phenomenon is that not all audit trails 
will be of the same length. 

3. TIkC problem of conceptual distance. 
This is a classical measurement prob- 
lem which surfaces in the Interpreta- 
tion of data from interactive treat- 
ments. Given different treatments, 
selections at nodes may represent 
nominal, ordinal or even integer data 
points, and each tjrpe of data imposes 
restrictions on how data can be ana- 
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lyzed. For example, one menu in a 
program ofiers the viewer a choice of 
"river memories," "river bridges," 
"river travel," or "river science." In 
this case "river memories" Is srepre- 
sented. as I and "river science" as 4. 
Because the data are nominal, the 
numerical representation is mislead* 
ing; the conceptual distance Implied 
between the two choices is in fact no 
greater than the conceptual distance 
between any two other choices from the 
menu. The problem is not isolated to 
numerical data. With tree diagrams 



and other graphic approaches, concep- 
tual distance Is implied as branches 
diverge on a diagram. In actual fact, 
however, choosing "A" at the seventh 
level of the farthest bianch on the right 
side of a diagram is not necessarily 
different conceptually from choosing 
"A" at the fourth level of the lea 
branch. Nevertheless, a casual 
observer can be seduced Into thinking 
that spatial or ntunerical distance in 
data indicate conceptual distance as 
well. 



Descriptive Approaches to the Audit Trail 



In our initial search for a meaningful 
way to represent the audit trail* we 
in^restigated and considered several 
formats. The list we generated, below. 
Is not exhaustive, but merely a point 
of departure. Some of the representa- 
tions appear to be more useful and 
durable than others: some we consid- 
ered brieffy and discarded for various 
reasons outlined below. 



RawDataMatrix 

The most basic w^ of representing 
audit trail data is to simply record the 
responses of each learner (as a vector), 
one above the other (see Figure 1). 
Matrices like these have the ad\^- 
tage of being easily constructed and 
relatively easily interpreted for each 
individual. The interpretation, howr 
ever, can onty be relatively limited, 
and context-bound. For crude forma- 
tive evaluation purposes, the ^ata are 
useful. An instructional designer can 
see which choices are attracting indi- 
vidual learners, and speculate about 
design decisions. But there is a seri- 



ous limitation with this type of 
approach: raw data buy definition, 
aren't simmiarized, and therefore 
conclusions based on the group are 
difficult to derive. 



Nodal Frequencies and Proportions 

Another approach we investigated 
was to present data associated with 
each decision point (node) in the 
treatment Data can be presented in at 
least two forms, as raw data (Figure 2) 
or as proportions (Figure 3). 

Raw nodal frequencies, like raw data ma- 
trices, are easy to create. They are perhaps 
easier to interpret, since the data are now 
summarized. Magnitudes of differences are 
obvious, and at aiqr node, comparisons 
have high precision and are intuitively sat- 
isfying. At the same time, relationships 
across nodes, or among variables are dim- 
cult to interpret. For example, how should 
four choices of "A" at one node be compared 
with 103 choices of "A" at another node. If 
they occupy different locations in the 
treatment? Perhaps only eight Individuals 
encountered the first node, whereas several 
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hundred encountered the second. The loop- 
ing problems, described earlier in the 
paper, surface here to cause difficulties in 
interpretation. If an individual loops 
through a node several times, frequency 
data become distorted. Either the same 
choice is made several times, thereby 
inflating the frequencies at that node, or 
several different choices are made, thereby 
levelling the data at that decision point. 

Nodal data can also be presented propor- 
tionally. Again, this type of foraMit Is easy 
to create and precision is retained at a higli 
leveL Proportions allow easier compar- 
isons across nodes or among variables at 
different positions in the treatment. Of 
course some calculation is necessary, and 
the user must struggle with the question of 
what to use as a denominator. For exam- 
ple, is the denominator consistently the 
total number of learners encountering the 
treatment, or is the denominator the total 
number of learners who pass a particular 
decision point? Perfiaps obviously, the 
denominator of any proportion will be 
determined by the comparisons the user 
chooses to make— yet another type of con- 
text dependence. As with raw nodal data, 
proportional data are also sensitive to 
looping problems. An individual looping 
several times through a particular node can 
inflate its proportion of the total, hi addi- 
tion, as one dec<:ends deeper into he data 
matrix, smaller raw numbers represent 
elevated proportions (see Figure 3). While 
proportional representation is useful for 
compressing large numbers, statistically 
bloating small numbers appears urmeces- 
saiy and counterproductive. 

When considering nodal data (either 
frequencies or proportions), individ- 
ual differences 5U"e lost in the com- 
pression of data. Any design deci- 
sions based on these data are limited 
to conclusions about the group as a 
homogeneous entity, and we sacrifice 
any more subtle interpretations. 
Furthermore, data spread across sev- 



eral tables, each representing a single 
node (or. alternatively one large table 
showing multiple crossbreaks). are 
difficult to assimilate. Patterns that 
exist within them are difficult to 
detect. 



PetitPobtt Pattern 

This early approach to the portrayal of the 
audit trail was suggested by son.e of John 
Tuke/s work in representing non- 
parametric data in what he called a "stem 
and leaT form (Hartwig & Dearlng. 1979). It 
is a graphical approach, combining the 
intuitive appeal of a histogram with a 
character-based notational system symbol- 
tdng the choices. For example, in Figure 4, 
the Xs represent the first choice. 0*s repre- 
sent the second choice, and H's represent 
the third choice. A dot (•) is used to indicate 
that the learner proceeded past the node 
without making a choice, a situation made 
possible by an tuifortunate bug in the pro- 
gram used to collect these sample data. 

Although vlrtualfy any symbol could 
be used to represent individual 
choices, but we cliose characters that 
seem to occupy the same amount of 
space, so that inter-character dLTer- 
ences would not influence the overall 
appearance, and perhaps the Interpre- 
tation, of the display. The resulting 
pattern of characters Is in some ways 
reminiscent of a pattern for petit- 
point embroidery. 

To set up a display of this kind, data 
must be progressively or sequentially 
sorted decision point by decision 
point. That is. subsequent columns of 
data in a matrix must be sorted within 
tl^e categories formed by the sorted 
data comprising the first colunm. 

Advantages of this method included 
the ease of generating the display by 
using a search and replace function on 
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a word processor to substitute charac* 
ters. and a line sorting feature to 
assist in constructing the display. 
Although the display accurately 
shows the proportions of choices 
made at each node in a graphic and 
intuitive way, the dependency prob- 
lem is wry evidently in play: In the 
fifth column, for example, there are 
11 distinct groups of 0*s: each group 
has a different meaning, depending on 
where it is located vertically. That is, 
although the 0*s all indicate that the 
second of the choices available was 
the one that was chosen, the first two 
0*s and the third O represent choices 
on different content, due to the fact 
that different routes brought the 
learners to he node. 

In one version of this kind of display, 
we also tried to use color to denote 
different choices, but found that it 
offered little advantage in inter- 
prctability. 

This approac i was eventually set aside 
because it dldn*t seem to do a great deal to 
describe what was happening. It was 
deemed to be moderate^ useful as a forma- 
tive evaluation tool, but dldn*t appear to 
have sufficient power to make it useful for 
research purposes. 



Audit Trail Tree 

The audit trail tree (Figure 5) was the next 
approach we attempted. It combined both 



the graphical representation and the 
numerical accuracy of the Petit-Point 
Pattern approach, but, in addition, pre- 
sented the data in a more Intuitively power- 
ful way. 

The audit trail tree is drawn so that the 
thickness of the line depicts the number of 
learners who chose the path represented. Of 
cr ursc. If large numbers of learners are 
involved, the line width could be scaled. 
Too, ntimbers (either frequencies or propor- 
tions) could be attached to each node to 
provide greater detail. The visual represen- 
tation appeared to be useful and somewhat 
easier to interpret than the Petit-Point 
Pattern method. It was clear, it was graphi- 
cal, it was Intuitive, and it was grounded in 
reality. Comparisons were easy to make: 
flow could be read into the diagram as 
learners progressed from the begirming to 
the end ctf instruction. Although the draw- 
ing process is not difficult to do manually, 
it is somewhat tedious, and automation of 
the process on a graphic-interface micro- 
computer should be reasonably straight- 
forvfard. 

On the other hand, unless numbers were 
attached (as suggested above), the precision 
of the display was fairly low (i.e., it is diffi- 
cult sometimes to tell the difference 
between 3 learners and 4, or between 1 1 and 
13). The problem of conceptual distance 
remained, and perhaps was magnified by 
the ease with which other dimensions of 
the data were made manifest. And. of 
course, the problem of how to represent the 
loops remained. 
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Inferential Approaches to the Audit Trail 



Inferential approaches would be used when 
comparisons are being made between 
groups of learners (e.g., differing on cogni- 
tive styles) or treatments (e.g., using 
instructors of the same gender as the learn- 
ers and using instructors of differing 
genders). 

One inferential approach to the audit trail 
we considered and discarded involves the 
use of multiple regression. Tills traditional 
approach to path anafysis did not appear to 
be appropriate for answering the types of 
questions we are addressing here. It is a 
method of analyzing linear relationships 
among sets of variables, and assumes that a 
causal order among the variables is known 
and that the relationships among the vari- 
ables are causally closed (Duncan, 1966). 
Even though we didnt give this approach 
much consideration, we mention it because 
of a possible confusion of terms: We are 
attempting to anafyze paths through 
instruction, in a way that beais no rela- 
tionship to path analysis, as the term is 
used in a statistical sense. 

Furthermore, in an inferential approach to 
analyzing choices made in hypermediated 
and interactive Instruction, one cannot 
make the assumption of nonnal distribu- 
tion that underlies parametric statistics: 
hence a focus on the non-parametrics Is 
essential. 



A productive approach to analyzing a class 
of problems such as that under discussion 
would be to collect data on choices made by 
a laige group of people, and regard that dis- 
tribution as the usual distribution (in fact, 
it would be an expected distribution thai, is 
"normal" for the particular content and 
treatment being investigated, but since the 
term normal distribution has a technical 
coixnotation, we must make a distinction). 
Given this expected distribution, one could 
then subject certain individuals to treat- 
ments of vaiying kinds, and compare the 
audit trails of those subje^'ts to the audit 
trails generated by the "usual" population, 
on a decision by decision basis (and keeping 
in mind the dependency problem). That is, 
the comparisons could only be made for 
single decision nodes at a time (which could 
be a limitation). 

A statistic such as the one-sample 
test, a test of goodness of At (Siegel, 
1956), would appear to be an appro- 
priate tool for determining the statis- 
tical significance of observed desola- 
tions from "usuality". Another likely 
candidate would be the Kolmogorov- 
Smimov test (Marascuilo & 
McSweeney, 1977; Siegel, 1956). 
Indeed, since Siegel states that "...the 
Kolmogorov-Smimov test may in all 
cases be more powerful than... the 
test" (1956, p. 51), it would seem to be 
the test of choice. 



Issues and Challenges 



Xhere are issues and challenges arising in 
both the descriptive and inferential appli- 
cations of audit trail analjrsis. 

From the point of view of descriptive appli- 
cations, an important issue underlying all 



the audit trail methods Identified above is 
this: How does one deal with representing 
the fact that an individual learner may well 
traverse a node more than once? On one 
hand it would be possible to design tracking 
devices which would ignore subsequent 
identical path choices once an individual 
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has chosen the first time. But looping 
through the same path more than once may 
represent Important data. How then can we 
discriminate betweejii multiple encounters 
with a single individual across a path, and 
several single encounters with individuals? 
This problem remains imresol^d, and 
deserves attention. 

The highly contextual nature of all analy- 
ses continued to plague us as we searched 
for analytical tools. It seemed like eveiy 
avenue we pursued was predicated on a 
thorougli analysis of the treatment under 
study. The content not onV determined to 
some extent which of the analytical tools 
were appropriate, but also limited our abil- 
ity to generalize our conclusions at this 
stage of investigation. Certainty, we were 
humbled by the levels of complexity which 
seemed to settle like river sediment on what 
we Initially thought was a simple notion. 

The potential magnitude of an audit trail is 
also a challenge. Experience with collect- 
ing audit trail data in a variety of CAI and 
hypermedia environments (M. W. Petruk, 
personal communication, February 7, 
1990) confirms the intuitive notion that 
extremely large datasets are possible. If one 
considers the potential number of choices 
available in an instructional sequence that 
involves any degree of learner choice, and 
factors in the possibility of looping through 
some or all nodes, the sheer si^e of the audit 
trails becomes daunting. Given that some 
learners will loop more than others, the 
audit trail data matrix will not be square: 
this in itself makes it awkward to work 
with. 

In the inferential approach involving 
or Kolmogorov-Smimov, four dis- 
tinct issues arise. The first has to do 
with the fact that aiQr research must 
necessarily be conducted by compar- 
ing learner's responses at onfy one 
node at a time. Ifthe number of nodes 
is large (as Is liable to be the case tn 
most Instruction that incorporates 



learner control) then there exists the 
potential for an inflated a, similar to 
the situation where repeated t-tests 
are employed (I^ran, 1972). Other 
than recognizing the potential prob- 
lem and selecting an appropriate a to 
compensate for it, no evident solution 
presents itself. 



Another issue that arises in conjunc- 
tion with the use of the or 
Kolmogorov-Smimov test: Which 
expected distribution should be used? 
Above, it was suggested that an 
empirically-derived expected distri- 
bution should be employed as the 
"usual". However, it could also be 
argued that a rectangular distribu- 
tion, based on the number of paths 
available at each node, is the distribu- 
tion of choice. More development of 
this argument, and identification of 
the pros and cons of using each distri- 
bution, are needed. 

If it turns out that the empirically- 
derived distribution is the one of 
choice, then this question arises: How 
large does the population describing 
that distribution have to be in order to 
be adequate? 

Finally, there is the opposite side of 
the coin: What should be the size of 
ai^ sample of learners used in empir- 
ical research, given the knowledge 
that with larger sample sizes, x^ is 
know to become increasingly sensi- 
tive? That is. given a laige sample 
size, the use of niay lead to incor- 
rect decisions to reject Hr> How 
Kolmogorov-Smimov behaves under 
these conditions is not clear. 

Another consideration is that the 
nature of the content or subject matter 
may determine subsequent analyses, 
and we have not yet addressed this 
issue. In procedures we have dis- 
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cussed, the data do not represent the 
shapes of the instruction encountered. 
For example, they do not describe 
whether the learner is engaged in con- 
cept acquisition, problem sohring or 
chaining at ai^ particular point. 
Tills does not ^sq^ matter, but 
where it is an important feature for 
analysis, the initial content anatysis 
will need to take this into account It 
is also possible that specific analyti- 
cal procedures will need to be devel- 
oped to address specific instructional 
types. 

If we are to draw a single set of conclu- 
sions from our investigation of this 
issue to date, we suggest that the 
search for a single appropriate ana- 



fytical tool for interactive media 
research Is futile. We believe a thor- 
ough ana^is of audit trail data 
requires a combination of techniques, 
both descriptive and inferential, to 
adevquately address research ques- 
tions. As our experience with existing 
tools grows, prescriptions concerning 
when to use combinations of 
approaches may be possible, but at 
this point, such prescriptions would 
be premature. Further, rather than 
viewing the context-dependence of 
each analysis as a barrier or limita- 
tion, we believe it is more productive 
to view content or task analysis as a 
prerequisite step to any further anal- 
ysis. 
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Figure 1. Sample raw data matrix. 
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Figure 2. Tabular representation of raw frequencies of paths chosen In sample data. Although the sample data actualty have eight 
decision nodes, onty four are presented here, for simplicity. Note how quickly the size and complexity of the table grows 
as the number of decision nodes and/or the number of choices available at each node increases. 
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Figure 3, Tabular representation of proportions of paths chosen in sample data for first four decision nodes. Note how. as one 
progresses through more decision nodes, smaller frequencies of responses produce deceptively larger proportions. 
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irs 4. Petit-Point Pattern method of representing an audit trail. An X represents the choice 
of the first possible path from a node, an O represents the second, and an H represents 
the third. A • indicates no response. 
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